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Issues and Strategies for Evaluation and Program Improvement 



Abstract 

In the 1990's, federal education legislation and policies have encouraged state and 
local efforts in educational reform and placed a new emphasis on education for and 
about work. These initiatives have engaged the active participation of a broad range 
of partners, and have spurred a variety of locally customized programs and practices. 
Nationally, Tech Prep and School-to-Work (STW) initiatives have been developed and 
implemented in a political climate that is increasingly devolving funding, 
responsibility and accountability to the local level. Evaluation of these efforts must 
meet local, regional and federal demands for information that address both 
accountability and program improvement needs. This paper presents a model for 
benchmarking that addresses some of these evaluation needs. The benchmarking 
model presented views the evaluation of Tech Prep and STW through the component 
practices which make up each initiative. This is in line with Wisconsin's approach to 
Tech Prep and STW as school reform efforts which apply to all students. At the 
heart of this model is a data collection system and self-assessment process intended 
to provide both local program improvement information and links to district, state 
and federal accountability and performance standards requirements. Initial pilot tests 
show the usefulness of this model as a tool \vhich increases teams of practitioners' 
understanding of the Tech Prep and STW practices within their school. Although a 
lack of common definitions of component Tech Prep practices, as well as institutional 
and attif^udinal barriers, limited the effectiveness of data collection, practitioners 
found the process to be l- beneficial tool for program improvement. 



The Evaluation Context 

Shifts in federal education policy lead to systemic change, multiple constituencies and 
increased local cutonomy. 

In the I990's, federal education legislation and policy, such as set forth in the 
Carl D. Perkins Act Title III E Tech Prep program (P.L. 101-392) and the School-to- 
Work Opportunities Act of 1993 (P.L. 103-239), have encouraged state and local 
efforts in educational reform and placed a new emphasis on education for and about 
work. Tech Prep and STW initiatives are being developed and implemented in a 
political climate that is increasingly devolving fundi ig, responsibility and 
accountability to the local level. In response, early state and local initiatives have 
implemented widespread systemic change, engaged the active participation of a 
broad range of partners, and generated a variety of locally customized programs and 
practices (Pauley, Kopp and Haimson, 1994). These comprehensive changes present 
an array of challenges for evaluation. The broad scope of both the Tech Prep 
initiatives and the School-to-Work Opportunities Act will require evaluation 
techniques that can bridge the gaps between education and employment, between 
new institutional relationships, and between unique school practices and long-range 
student outcomes. In the following section, the federal education policy context will 
be discussed in light of the challenges it poses for evaluators, educators and policy 
makers seeking to understand and evaluate these new education initiatives. 

In the 1990's, an emerging shift in federal education legislation is driving a 
'"«roader framework for systemic education reform. Federal legislation is moving 
toward funding more diverse implementation strategies for educational programming 
for all students at the local, state and federal level. In a report considering the 
reauthorization of the Elementary and Secondary Education Act, the GAO 
summarized the nationwide movement toward system-wide educational reform 
(GAO, 1993). Driven by the development of n rnnge of local, state and federal 
standards, systemic educational reform efforts are gaining momentum. For example, 
the Goals 2000: Educate America Act (P.L. 103-227) is based on principles learned 
from successful school reform efforts. The act supports "bottom-up" approaches to 
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school reform by encouraging schools to voluntarily adopt occupational, academic 

and opportunit)^-to-leam standards. The act also encourages local involvement in 

program development and stresses local and state participation in comprehensive 

program improvement plans (U.S. D.O.E., 1994). 

The recently enacted School-to-Work Opportunities Act of 1993 (STWOA) also 

reflects this shift in the mission of the nation's public schools. STWOA calls for a 

fundamental restructuring of the educational delivery system, providing a loose 

framework for organizing state and local initiatives around school-based, work-based 

and connecting activities. The legislation promotes school-to-work transition as a 

systemic initiative, grounded in local partnerships which are encouraged to customize 

school-to-work programs to fit local needs and resources: 

States and localities will take the lead in determining goals and 
priorities, developing new strategies, and measuring progress 
(Federal Register, October 1993, p. 53388). 

These rhanges are taking place in a political climate that is increasingly devolving 
funding and responsibility to the local level, increasing local autonomy and providing 
seed-capitol for the development of locally managed educational systems. 

The breadth of STWOA requires extensive participation across institutions, 
among partners and over time. The legislation encourages the formation of local 
education and training systems comprised of partnerships among secondary and 
postsecondary educational institutions, private and public employers, labor 
organizations, government, community groups, parents and other key stakeholders. 
This educational in.tiative expands the institutional boundaries of schools and calls 
for a dramatic shift in the roles of teachers, administrators, guidance counselors, 
parents and students. 

The act challenges educators to intep;rate academic and vocational instruction, 
link school- and work-based learning, step up career development activities, and 
align secondary and postsecondary educational pathways. Educators will be expected 
to assume responsibilitv lor the economic futures of students (Berryman and Bailey 
1992), and provide a more rigorous career- and technologically-oriented education to 
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students than what is currently the norm (Phelps, 1992). Similarly, the act will 
challenge business and community members to assume more extensive roles, beyond 
simply serving on advisory committees (Phelps, 1995). Business, labor and 
community members will need to form effective partnerships with educators in order 
to provide instruction in workplace skills and broad aspects of industry; as well as to 
lend their expertise in developing skill standards, credentialing and mentor training. 
All of these roles are required under the new School-to-Work Opportunities Act. 

STWOA is not the only piece of federal education legislation and policv 
placing increased attention on collaboration among service providers, across agencies 
and between legislative initiatives. Tech Prep, as authorized under the Carl D. 
Perkins legislation, also requires collaboration between secondary and postsecondary 
schools in establishing articulation and the necessary linkages for curriculum 
alignment and sequencing of programming. Under this act, vocational education 
programming in general is expected to coordinate locally with other federal education 
and training programs such as JTPA and provide equal access for all students to the 
full range of programming and services. 

Both the Tech Prep and STW initiatives are far-ranging, and require a complex 
network of programs and practices which will link the education and employment 
svstems. These reforms require the participation of new publicXprivate partnerships 
which will increase the information demands from multiple constituents with 
divergent needs. These complex reforms will require systemic development over 
time, with local programs evolving as resources and expertise are acquired to 
implement them. This broad context creates conceptual, organizational, and political 
challenges for evaluators who must contend with multiple sets of goals, philosophies, 
and information needs. Because School-to-VVork is a relatively new initiative, 
guidelines and procedures for accommodating these multiple needs within evaluation 
have not yet been set forth. Many of these same problems must also be faced when 
attempting to examine the development and impact of Tech Prep initiatives. 
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New initiatives place additionaL demajids on evaluation: the challenge of accountability and 
program improvement 

Increasingly, educators and policy makers are requesting evaluation feedback 
that informs both accountability and program improvement. There is an increased 
demand to address accountability both in service delivery (by monitoring program 
implementation and practices), and program impact (by gathering and analyzing 
student outcome data). Additionally, educators need information on program 
processes and practices to help them improve on current efforts and to meet new 
standards in education. This increased pressure for accountability coupled with the 
simultaneous demand for school- level information for piogram improvement 
suggests the need for evaluation and planning models which link data with program 
improvement. Addressing these needs is particularly important when assessing long- 
term educational reform initiatives which are both complex in their processes and far- 
ranging in their goals and intended outcomes. 

An example of this dual demand for accountability and program improvement 
information is set forth in the Carl D. Perkins legislation. The 1990 Reauthorization of 
the Carl D. Perkins Act mandated significant changes in federal vocational education 
policv by requiring an outcome-based accountability system. Outcomes (e.g., 
academic and occupational skill attainment, program completion ana employment) 
include both student and program accomplishments. The re-authorized act places 
most of the burden of accountability on the states, while allowing a great deal of 
flexibilitv to the states in individualizing performance measures and standards. The 
act also requires local partnerships to be responsible for program improvement, and 
to submit local program improvement plans if the state standards are not met. The 
;Njati.onal Assessment of Vocational Education report states that while states have 
made progress in developing performance standards and measures, many localities 
have experienced difficultv implementing performance measures and standards and 
coherentlv linking; these outcomes to local program improvement (U.S. D.O.E., 1994, 
p. 285). Effective linkage of vocational education evaluation and planning has been 
frustrated bv a lack of models linking outcome-based performance measures to local 
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program improvement planning and strategies (Stecher, et al., 1994). 

While there is a need to improve the implementation of existing performance 
measure systems for student and program performance outcomes at the state and 
district levels (Stecher, et al., 1994; Willms, 1994), there is also a need to complement 
this data with specific information on individual schools and their programs 
(Salganik, 1994). Evaluators and researchers have long emphasized that school-level 
indicators are needed for measuring prograra success because they account for 
school-level differences and allow for like-school comparisons. In addition, qualitative 
and quantitative program information, collected and analyzed locally, can better 
inform school improvement by enhancing local practitioner involvement and 
understanding (Porter 1993b; Holcomb 1991; Newman, 1991). 

A recent rise in the number of schools implementing site-based management, 
embarking on significant school restructuring and reform efforts and using TQM 
processes for local program improvement, is also changing the local context for 
evaluation. The nature of these reform efforts requires that schools take responsibility 
for local program planning, development, implementation and assessment. This 
means that individual schools must assess the quality of their educational processes 
and practices, and take primary actions for program improvement. As a result, there 
is now an increased need for locally relevant data collection to help inform these 
processes (Phelps, 1991). 

A review of the literature on educational indicators reveals a trend endorsing 
the development of both a set of school-specific process and delivery measures 
(performance measures) and a set of acc(umtability and outcome measures, in order 
to fully understand school reform and change (Stecher, et al, 1994; Wholey, Hatry, & 
Newcomer, 1994; Porter, 1993a; Phelps, 1991). Allen Phelps (1991) suggests that a 
comprehensive and viable performance assessment system should: 

1. provide administrators and instructors with information that is useful in 
improving programs; 

2. enable students, parents and others to understand clearly the purposes, 
intended outcomes and effecti\'eness of the programs; and 
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3. provide information for use in determining accountability at the local and 
state levels. 

In an article discussing the development and use of school delivery standards, 

Andrew Porter supports the need for process standards that are neither minimum 

standards nor prescriptions. He suggests that school performance standards leading 

to a system of school input and process indicators should be locally controlled, 

encouraging teachers and administrators to accept responsibility for local inputs and 

procedures (Porter, 1993a, p. 29). He states that: 

Hie indicators would provide useful descriptions of school practices and useful 
insights into areas needing attention and how they might be addressed. 

Porter writes that school process standards are needed to "create a vision for what 
education can be" and that accountability should be measured as the "school's value- 
added student performance" (Porter, 1993a, 29). By coupling school delivery 
standards with student performance accountability, schools will be better prepared to 
anticipate and adapt to change and educational reform. Porter concludes: 

Whatever strategies and solutions are put in place today must anticipate 
change. If the history of education reform tells us anything, it tells us 
that commitment to continuous improvement in education is 
necessary.... We must build a system that has the requirement to 
monitor itself and the capacity to change when necessary. (Porter, 1993a, 
p. 29) 

These two-fold pressures for accountability and program improvement 
information come at a time when education is poised for the adoption of complex, 
educational reform initiatives such as proposed in the School-to-Work Opportunities 
Act. Clearly, the scope of both Tech Prep and School-to-Work, and the accountability 
and program information needs of educators and policy makers, present unique 
problems for evaluation. In particular, evaluators of these initiatives will need to: 

1 ) coordinate the evaluation needs of participating education, business, labor 
and government institutions; 

2) understand the connections between school-based practices and student- 
based outcomes; and 

3) provide locallv relevant infcM-mation meaningful for program improvement. 
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The benchmarking model presented in the following sections proposes to address 
many of these challenges facing evaluation of such educational reform initiatives as 
School-to-Work and Tech Prep. 

Adapting a Benchmarking Model for Education 

The benchmarking model presented in this paper seeks to begin the process of 
addressing the challenges faced by evaluators, educators and policy makers in 
addressing both accountability and program improvement, while meeting the 
information needs of multiple constituencies. The benchmarking model is adapted 
from tools and processes for continuous program improvement used in the business 
sector. The model is intended to be a framework for school self-assessment and data 
collection which feeds into the school plarming process and applies continuous 
improvement strategies. The model is not intended to be a complete evaluation 
system; rather it is a process which, through the use of a set of tools, promises to 
address the wide range of information needs of evaluators. 

The research activities associated with designing and testing a benchmarking 
system for Wisconsin's School-to-Work and Tech Prep initiatives provide the basis for 
this model. . 'rior to designing the benchmarking model, project staff reviewed 
literature on performance measures and standards, \'Ocational education 
programming, and school and student indicator syt^vims (e.g.. Barton, 1994; Porter, 
1993a). Additionally, project staff studied literature on the Effective Schools process 
(Rossmiller & Holcomb, 1992; Holcomb, 1991; Newman, 1991), Total Quality 
Management (TQM) in educational settings (e.g., Rinehart, 1993; Schenkat, 1993; 
Bonstingl, 1992; Bragg, 1992; Spanbauer, 1992, 1987), benchmarking (McNair & 
Leibfried,1992; Spendolini, 1992; Camp, 1989 ) and the use of benchmarking in 
education (Seymore, 1994; Inger, 1993a; Marchese, 1993). Loc.Jy relevant 
information was gathered from Wisconsin state policy reports, program applications 
and legislation. Stakeholders were consulted through independent interviews with 
agency staff, project advisory meetings with school administrators and focus group 
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sessions with in-school practitioners. The resulting benchmarking model, while 
specific to the state of Wisconsin, offers lessons for other states and schools 
attempting to use this process to evaluate and improve similar initiatives. 

Prior to developing a benchmarking model for educational settings, project 
staff reviewed and adapted benchmarking models and practices developed by and 
for business and industry. 

In the business sector, benchniarking is a process that begins with self-study. 
It involves the systematic collection of internal data and information which allows the 
business to compare its own processes with similar processes used in other 
organizations. Information, ideas and data obtained from these other organizations 
contribute to planning and program improvement in a continuous improvement cycle 
(McNair & Liebfried, 1992; Spendolini, 1992; Camp, 1989 ). 

The practice of benchmarking promises to disclose gaps between perceived 
and actual performance, and overcome complacency by exposing inaccurate 
perceptions. The benchm.arking process accomplishes this through exhaustive internal 
self-analysis of processes, practices and data collection. The data collection serves to 
establish a baseline from which to compare with other organizations, and provides a 
basis for setting localized targets against which to measure performance 
improvement. The self-assessment of practices and processes, combined with data 
collection and analysis, allows for in-depth and meaningful comparison with other 
organizations. The comparison between organizations is a source for innovative 
ideas for correcting (or even eliminating) problems and uncovering emerging 
practices that can focus future efforts towards continuous improvement (Economist 
Intelligence Unit, 1993; Inger, 1993a). 

While benchmarking is a relatively new concept, many lessons have already 
emerged from those businesses using the benchmarking process. First, it is essential 
to include and meet the n( -ds of all stakeholders when designing and conducting a 
benchmarking investigation (McjNnir & Liebtried, 1992; Spendolini, 1992). Second, 
succes^sful benchmarking involves identifying core issues (McNair & Liebfried, 1992) 
and focusing on specific processes and program improvement needs (Spendolini, 
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1992). Attention to thorough self-assessment and maintenance of relevant data for 
follow-up are also keys to benchmarking success (Spendolini,1992). 

In designing a benchmarking process for schools, several organizational and 
institutional factors which differentiate schools from businesses were noted. The 
underlying incentives for business to improve their processes and programs are 
inherently different than those faced by schools. In business, the "bottom line" is 
profit ~ increasing the profit margin in a highly competitive market requires focusing 
on customer satisfaction and by improving efficiencies in production. For schools, 
the focus is on providing quality education for students. However, schools are hard 
pressed to idenhfy a distinct "customer"- education is considered to be a public 
good, a benefit to all, including students, parents, community members and 
employers. Each of these "customers" have unique demands, and improving 
educational satisfaction requires that broad range of educational outcomes be met. 
Furthermore, information on business products and efficiency is easily quantified, 
while information on quality and efficiency in education is not. 

As noted earlier, broad educational initiatives such as School-to-Work and 
Tech Prep consist of a variety of locally customized approaches. The benchmarking 
process is ideally suited to analyze these local efforts because the process of 
benchmarking focuses on self-assessment of "components," or practices and 
processes. The benchmarking process allows for self-assessment of locally developed 
components and practices while encouraging data collection relevant to the 
improvement of these activities. It is therefore not necessary to establish an 
identifiable "product," as the unit of analysis is component-based. 

It should be noted that the benchmarking model is less of an evaluation 
model than a program improvement strategy which incorporates such evaluative 
concepts as problem assessment and data collection. The use of this strategy assists 
evaluation bv creating a climate where data collection and evaluation are seen as a 
natural part of program development. In such a climate, practitioners are more likely 
to take ownership of data collection and evaluation procedures, increasing the 
likelihood of valid and reliable data collection. Data is seen not as a threat, but as a 
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tool. 

In adapting these facets of the benchmarking process for use in an educational 
setting, attention was placed on developing a set of program components, and both a 
process and a set of data collection tools, to provide information on those component 
areas to a variety of stakeholders. The following sections will detail the benchmarking 
model designed for use in the Wisconsin Cooperative Tech Prep Benchmarking 
Project. 

Wisconsin's Cooperative Tech Prep Benchmarking Model 

The model presented below represents an irmovative approach to the use of 
benchmarking lo bridge the gap between accountability and program improvement. 
This model was originally developed through a project funded by the Wisconsin 
Department of Public Instruction, Wisconsin Technical College System, and Wisconsin 
Council on Vocational Education to establish a benchmarking system for Tech Prep in 
Wisconsin. Tb's project was designed to meet the needs of a number of 
constituencies, including both state-level agency representatives and local 
practitioners. To do so, a number of goals were developed: 

• Validate existing performance measures and standards; 

Validate the existing "Quality components" which make up Wisconsin's 
goals for Tech Prep; 

Define Tech Prep as it exists in Wisconsin based on the practices actually in 
place; ai.d 

• Establish a benchmarking system which would accommodate the needs of 
state policy makers and local practitioners. 

The structure of Tech Prep and STW in Wisconsin necessi*-ated taking an novel 
approach to both benchmarking and evaluation. In Wisconsin, Tech Prep is generally 
seen as a school reform efforts, not as n distinct program. Each is a set of educational 
processes and practices applied across the curriculum. For this reason, there is no 
defined cohort of "Tech Prep Students." Rather, Tech Prep represents strategies used 
with all students. This presents difficulties for evaluators wishing to define and 
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follow such a cohort of students. Furthermore, there is no set of standardized Tech 
Prep strategies applied among Wisconsin schools. Instead, secondary schools are 
loosely tied together in a number of consortia, each of which has been developed 
around a technical college. This has lead to the development of a wide range of 
different approaches to Tech Prep. There is little agreement on what constitutes Tech 
Prep, and no consistency among the definitions guiding the different practices 
brought together under the Tech Prep initiative. 

Benchmarking the Implementation of Educational Practices 

Because there is no cohort of Tech Prep students in Wisconsin, this 
benchmarking process focuses on the quality and nature of educational practices. This 
represents a shift in the unit of analysis from the student to the implementation of 
educational practices. The student is considered within this process; however, the 
focus is on the level of implementation of a given set of practices: whether or not a 
given element of Tech Prep is in place, the level of implementation of that element, 
the rate at which students participate in it, and the rate at which desired outcomes 
are experienced by these students. 

This focus on the level of implementation of a given set of practices is also 
necessitated by the fact that Tech Prep differs from one Wisconsin school to another. 
Local systems are free to implement these practices in the way that best fits local 
needs. Focusing on practices and organizing these practices into a taxonomy of 
component areas provides the flexibility necessary to accommodate the diverse nature 
of Tech Prep as implemented by these many different local systems. 

Seven Component Areas of Tech Prep 

Seven component areas of Tech Prep were identified through focus groups and 
input from project stakeholders. Participants in these focus groups were asked 
indi idually to list all of the elements of Tech Prep necessary for a successful 
program. Together they sorted these elements into groups of common practices and 
gave each group a summative title. Project staff then examined and refined these 



12 



groupings in light of the existing literature on successful Tech Prep (e.g., Bragg, 
Kirby, Puckett, Trinkle, & Watkins, 1994; Bragg & Layton, 1994; Pauly, Kopp, & 
Haimson, 1994; Illinois State Board of Education, 1991; Carl D. Perkins Applied 
Technology and Education Act Amendments, 1990, P.L.101-392). The resulting 
taxonomy of Tech Prep components follows. 

• Access 

• Articulation 

• Career Development 

• Collaborative Partnerships 

• Curriculum Development 

• Professional Development 

• Work-based Learning 

These component areas provide the foundation for this model, giving it the flexibility 
needed to adapt to the many different ways Tech Prep has been implemented. This 
taxonomy also provides a powerful shorthand for practitioners to discuss the many 
and varied aspects of Tech Prep development. Such a shorthand helps categorize 
and define what Tech Prep is, easing the difficulties inherent in communicating about 
these diverse practices. 

Benchmarking Tools 

The seven component areas also provide a framework which links the different 
tools developed for the Tech Prep benchmarking process. Two categories of tools 
were developed for use with this model: self-assessment tools and data collection 
tools. Both tvpes of tools help practitioners focus on what they are doing in Tech 
Prep; identifv strengths, gaps, and problems; decide which areas need improvement; 
and determine whether changes they make are effective. 

Sdf-Asst'ssmcnt Tools 

The primarv self-assessment tool is the Tech Prep Implementation Checklist. 
The checklist provides a team of practitioners with a framework for examining and 
discussing their perceptions of the present level of implementation within each of the 
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seven component areas in their school. This checklist presents practitioners with a 
number of questions about their implementation of different elements of Tech Prep. 
For example, a question about career development asks: "Do all students have access 
to national, state, and regional labor market information?" Through group 
discussion, the team arrives at consensus about the extent to which their school has 
implemented the practice addressed in Mie question. In addition, they are free to 
state that a given element is not appUcaole to their local Tech Prep philosophy. 
These questions also act as a staff development tool, pointing out areas where the 
team may need to do some planning and vuviding an overall sense of the possible 
breadth and quality of Tech Prep program. ;.mg and activities. In this way, the 
checklist serves a programmatic function without being threatening or didactic. 

Data Collection Tools 

Information about the extent of implementation, participation, and outcomes of 
Tech Prep is gathered using a set of quaiiiative and quantitative benchmark measures 
(See Figure 1). Aligned with both federal and Wisconsin specifications for quality 
Tech Prep education, these benchmark measures provide a mechanism for assessing 
both the level of implementation and the quality of Tech Prep practices at the local 
level. Each comoonent area is examined in terms of implementation, student 
participation, and the outcomes expeuenced by those students. Implementation 
benchmarks are either qualitative descriptions of the practices within a component of 
Tech Prep or STW, or a quantitative measure of the extent to which Tech Prep 
practices are available within the school. Where implementation benchmarks reveal 
the quality of Tech Prep implementation, participation benchmarks indicate the 
depth. Participation benchmarks reflect the percentage of students and teachers 
involved in each Tech Prep component. Outcome benchmarks are used to measure 
the efficacy of each Tech Prep component. These measures are based on the 
acknowledged goals of each Tech Prep component, and are generally focused on 
what happens to students after high sch(M"il. Furthermore, the outcome benchmarks 
for each component area are specifically tied to that area's implementation and 
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participation benchmarks. This allows for an in-depth examination of the 
effectiveness of varying levels of implementation of and participation in specific Tech 
Prep practices. 

Bundled with these benchmark measures are recording sheets and data 
collection tools designed to assist practitioners in collecting data in a reliable and 
valid way. Preliminary investigations into practitioners' views of data collection 
made during focus groups indicated that many schools lack capacity and/or are not 
institutionally prepared to collect and make use of data. These data collection tools 
were designed with this in mind. In most circumstances, practitioners completing 
these data collection forms are asked to list individual courses, strategies, or events, 
to provide some basic qualitative program descriptions, and to mdicate the number 
of people participating in each (See Figure 2). Rarely are they asked to list aggregate 
data. Whenever possible, common and /or important elements have been included as 
check boxes to facilitate reliable reporting. The goal of designing the data collection 
in this way was to allow practitioners with little or no experience collecting and using 
data to experience success doing so. 

In line with this, a great deal of attention was paid to making these tools as 
user-friendly as possible. The data collection tools were designed to be useful in and 
of themselves, providing a place where practitioners could organize a wide range of 
necessary information. They were also designed to align with existing state-level 
accountability systems, particularly the performance measures developed by 
Wisconsin's Tech Prep State Management Team under Title IIIE of the Perkins Act. 
This ser\'es to ease the data collection burden on schools, as they can collect the same 
data for different purposes. 

Through the use of both types of tools, self-assessment and data collection, 
teams of local practitioners are able to focus their attention on specific aspects of Tech 
Prep in their school or district. The goal is to identify the level of implementation of 
specific, existing practices as veil as identify and describe problems or gaps at a 
manageable, tangible level. Tliis allows practitioners to move beyond rhetoric and 
anecdotal information and determine the underlyir'g factors behind both their 
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successes and their problems. In addition, this level of specificity is required when 
the teams seek out others' "best practices." When attempting to discover how others 
have successfully implemented a particular practice or solved a problem, questions 
which remain at a general level are likely to yield answers which will not be of any 
assistance. Finally, this process of narrowly defining areas to work on keeps the 
problems to be solved to a manageable scale, thus increasing the likelihood that 
practitioners will experience success with the process and continue to use it. 

Once the team members have an understanding of where they are in the 
implementation of Tech Prep as well as an understanding of some of the gaps and 
problems they face, their next step is to seek out others who have successfully dealt 
with such implementation. These best practices then become a source of ideas, 
information, and strategies which can be adapted to meet local needs. As these 
strategies are adopted, continued reassessment of the existing level of implementation 
will help monitor the success of these strategies as well as indicate directions for 
future attention. 

The remaining sections of this paper describe a pilot test of this model wiui six 
schools engaged in the implementation of Tech Prep. In these sections, the 
operationalization of this model in terms of specific tools, processes, and activities 
will be discussed, as will the results of this pilot test. The final sections of this paper 
will summarize the lessons learned from this pilot test and suggest directions for 
future development and research. 

Pilot Test 

In order to validate this model of the benchmarking process, six schools were 
recruited as pilot test sites. Suggestions for possible pilot site schools were elicited at 
a state-wide gathering of Tech Prep practitioners. Particular emphasis was placed on 
schools which had made significant progress in the implementation of Tech Prep. 
From these suggestions, six schools were chosen to represent a cross section of 
Wisconsin high schools in terms of size; rural, suburban, or urban location; and 
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willingness to participate. The four technical colleges affiliated with these six schools 
were also recruited. Table 1 Lists these schools, their sizes and location, and technical 
college affiliations. 



High School 


Size 


Location 


Technical College 


School A 


1176 


Urban 


TC 1 


School B 


1186 


Suburban 


TC 2 


School C 


1569 


Urban 


TC 3 


School D 


200 


Rural 


TC 3 


School E 


345 


Rural 


TC 4 


School F 


520 


Rural 


TC 4 



Table 1: Pilot Site Schools 



Once each pilot site was identified, a contact person at each site gathered 
together a team of local stakeholders, including administrators, teachers, and 
community members. A representative of the technical college was also invited to 
participate on this team. In some cases this team was identical to the local Tech Prep 
coordination team. These teams ranged in size from seven to fifteen people. 

Pilot Site Visits 

The pilot test consisted of two site visits and a networking session. In 
addition, pilot site personnel were asked to collect data on the implementation of 
Tech Prep in their school. Materials and processes specifically designed to facilitate 
the benchmarking process were used at each site visit. These materials and processes 
will be discussed as they were used in the pilot test. There were two facilitators 
present at each visit. These facilitators directed activities, answered questions, and 
made observations of the process. 
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Site Visit One 

The first site visit represented an opportunity for the local benchmarking team 
to come to a common understanding of what Tech Prep was in their school and the 
nature and extent of their implementation of Tech Prep. This first visit lasted eight 
hours, and was divided into three parts: Identification of local critical success factors, 
completion of the' Tech Prep Implementation Checklist, and discussion of the school's 
perceived strengths and weaknesses. 

Critical Success Factors. In order to focus team members' attention on Tech 
Prep implementation and bring to light diverse views of what Tech Prep is and what 
it should accomplish, team members were asked to brainstorm answers to the 
following four questions: 

1. What are the most critical factors for Tech Prep success? 

2. What specific operational problems have been identified in your school? 

3. What external pressures are being felt by your school? 

4. Which areas have the greatest room for improvement? 

Each of these questions was considered separately. The affinity process was used to 
generate a list of answers which included input from all team members. 

The Tech Prep Implementation Checklist. The Tech Prep Implementation 
Checklist was designed to offer team members a way of structuring the investigation 
of their current level of Tech Prep implementation. The checklist consists of a 
number of questions arranged according to the seven component areas, with a 
number of "subcomponent areas" within each component. These component and 
subcomponent areas are listed in Table 2 below. 
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ACCESS 

Recruitment/ Enrollment 
Assessment 

Support Services Related to Tech Prep 
Classroom Accommodations Related to Tech Prep 
Inclusive Approaches/Strategies 

ARTICULATION 

Articulation Agreements 
Transcripted Credit/Dual Credit 
Advanced Standing Credit 
Postsecondary Linkages 

CAREER DEVELOPMENT (EXPLORATION & PLANNING) 

Career Exploration 

Curriculum Map/Career Map 

Career Major 

Career Plan 

Career Portfolio 

Career Development Activities 

Job Shadowing 

COLLABORATIVE PARTNERSHIPS 

Tech Prep Coordinator 
Active Participation/Contribution 
Collaborative Partnc-- 
Strategic Planning 
Communication and Marketing 

CURRICULUM DEVELOPMENT 

Integrated & Applied hist ruction 
Integrated & Applied Content 

PROFESSIONAL DEVELOPMENT 

Faculty Internships /Job Shadowing Opportunities 
Joint Inservice/Staff Development Opportunities 

WORK-BASED LEARNING (WBL) 

Work-Based Learning 



Table 2: Implementation Checklist Components and Subcomponents 

Using existing literature (e.g,. Bragg, Kirby, Puckett, Trinkle, & Watkins, 1994; 
Bragg & Layton, 1994; Pauly, Kopp, & Haimson, 1994; U.S.D.O.E., 1994), practitioner 
and agency-kn-el focus groups, and fede-al and state legislation as a foundation, the 
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checklist questions were developed to reflect a consensus view of quality Tech Prep 
education. Each question asked about a specific aspect of Tech Prep such as, "Do all 
articulation agreements list the postsecondary competencies to be achieved in each 
articulated high school course?" Team members responded that this element of Tech 
Prep was either "In Place, " "Implemented," or "Planned." Team members could also 
indicate that they "Should Plan" a given element, or that an element was "Not 
Applicable." This last response was included tc give team members a way to 
indicate that an element of Tech Prep as listed on the checklist did not align with 
their local intentions for Tech Prep development. 

Team members progressed through the checklist, question by question, as a 
group. Discussion was encouraged for each question, although time limitations 
necessitated that the facilitators keep this discussion fairly focused. In order to 
answer a question, the team had to reach consensus on the level of implementation of 
that element of Tech Prep. In doing so, it was expected that team members would 
contribute knowledge about their own Tech Prep activities and learn about those 
aspects of Tech Prep outside of their own experience. 

Strengths and Weaknesses. One goal of benchmarking is to focus discussion 
on one tangible aspect of the process under consideration. To do so, members were 
asked to identify a component area of strength and an area needing improvement. 
This was done using the information gained through the Critical Success Factors and 
Implementation Checklist activities. In cases where the group could not come to 
consensus, the nominal group process was used to vote on the areas of strength and 
improvement. 

One ■ the area of strength and the area needing improvement were identified, 
the team was divided into two groups. One group focused on the area of strength 
and the oiher focused on the area needing improvement. Each of these two groups 
was asked to brainstorm what aspects of that area were already in place and what 
aspects of that area were missing. This served to further focus the team on the 
specific, tangible elements and practices in\'olved in Tech Prep. Following this, the 

or 
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whole team met once more, and each group shared their lists. The whole team was 
then given the opportunity to add to any of these lists. The site visit ended with 
team members completing a survey designed to give formative feedback on that 
day's activities. 

Data Collection 

Following the first site visit, contact people at each school and technical college 
were asked to complete a set of data collection forms. Based on the Tech Prep 
Benchmarks listed above, these data collection forms provided a standardized way to 
collect meaningful comparison information about each school. This data would 
supplement the information gained from the Implementation Checklist and would be 
used to provide a detailed picture of the depth of Tech Prep implementation and 
student participation at each school. 

This data, combined with responses to the implementation checklist, was used 
to create a Tech Prep Implementation Profile for each school. This profile listed each 
school's benchmark data for each Tech Prep componei '., as well as demographic 
information which could He used for ensuring meaningful, like-school comparisons. 

Site Visit Tioo 

Where site visit one represented an opportunity for team members to examine 
broadly the implementation of Tech Prep in their schools, site visit two gave them the 
opportunity to focus in on the specific area needing improvement identified in the 
first visit. This four hour session began with a review / of the results of the first visit's 
activities. Team members were then given copies of their school's implementation 
profile, which they reviewed and commented on. Next, team members reviewed the 
list of what was missing from their area needing improvement and identified gaps. 
After a short discussion in which members could add additional elements, team 
members used the nominal group process to prioritize these missing elements and 
selected two for further analysis. 
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Team members were then divided into two groups, with each group given 
responsibility- for one of the missing elements. Each group was given the goal of 
brainstorming a list of the gaps and barriers which have prevented this missing 
element from being implemented. Once this list had been created, each group 
developed specific questions it could use to find out how other schools had overcome 
those gaps and barriers. The goal of this exercise was to get team members to look at 
a specific aspect of Tech Prep in their school in detail and generate questions that 
could be used to guide their discussions with other schools during the networking 
session. Once both groups had finished generating a list of questions, their lists were 
presented to the whole team for discussion. As in the first, this visit ended with 
team members completing a survey designed to give formative feedback on that 
day's activities. 

Networking Session 

The goal of the networking session was to bring the teams from all six schools 
together to share their insights into the implementation of Tech Prep, reflecting on 
the checklist results, implementation profiles, areas of strength and improvement, and 
the questions generated in the second site visit. This meeting was held at the 
University of Wisconsin-Madison, and lasted four hours. Five of the teams were able 
to attend the meeting; the sixth was unable to attend due to inclement weather. 

During this session, teams were given two opportunities to network with their 
peers. The first was unstructured, which allowed them to get to know each other 
and find out about each school's strengths in Tech Prep. To facilitate this, each 
school was asked to bring examples of materials they had developed. These 
materials were arranged on tables around the meeting room so that team members 
could examine them. 

The second networking opportunit^' ^vas more structured. Each school was 
assigned a table, and ^.hn^ ,.me team members sought answers to their questions 
about their area need^.g improvement, others remained at that table to answer 
questions from other schools about their nrea of strength. The questions served to 
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facilitate discussion, and directed their conversations toward detailed, meaningful 
strategies for implementing and improving Tech Prep. 

After this second networking opportunity had taken place, participants 
returned to their teams and collated the information they had gathered. To assist this 
they were given a number of "strategy summary sheets" which asked them to list 
each strategy, its source, and the pros and cons of implementing it in their own 
school. These sheets were designed to facilitate the development of action plans, thus 
linking the information gathered through the benchmarking process to each school's 
planning processes. Each team filled out a number of these sheets as members 
shared the strategies they had found. 

Following instruction on how to develop action steps from these strategies and 
how to report their findings to their school's planning teams, participants were asked 
to complete two surveys. The first asked for summative information about their 
perceptions of the benchmarking process and their opinions of the impact it would 
have on their Tech Prep efforts. The second survey gathered their opinions on the 
use of data in their planning and decision-making processes. At this time 
participants were also given directories listing contact information for other pilot 
participants in the hope that they would continue the networking relationships they 
had started. Following this networking session, summary reports on each school's 
findings were developed and distributed to participants. Continued technical 
assistance was also provided as requested. 

Results from the Pilot Test 

This discussion will focus on participants' views of the benchmarking process, 
rather than on the implementation of Tech Prep in any or all schools. Combining 
responses from the four surveys indicated above with facilitators' observations yields 
a comprehensive picture of the strengths and weaknesses of this process. To 
structure this discussion, observations concerning the benchmarking process will be 
separated from those concerning data collection. 
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The Benchmarking Process 

On the whole, participants found this process to be both engaging and 
rewarding. When asked to rate the usefulness of this process on a seven-point scale, 
83% rated it 5 or better. The mean response was 5.2. Areas in which they thought it 
was most useful were planning and program improvement (30%), determining 
current program status (30%), and focusing the group and individuals on priorities 
(20%). 

When asked to rate the impact of this process on classroom practice, however, 
they were less positive. Again using a seven-point scale, 26% rated it 5 or better. 
The mean response was 4.0. Barriers to this process having an impact on classroom 
practice include: a need to extend the process and resulting information to all staff 
(247o), the process requires a lot of time and effort (20%), and a lack of administration 
follow-up or support (12%). On the positive side, 20% felt that the process developed 
greater awareness and buy-in on the part of staff. 

While participants were unsure of the impact this process will ultimately have 
on classroom practice, a majority (67". ) felt that they would continue to use it to 
further their implementation of Tech Prep. No respondents said that they would not 
continue to use this process, and 17% were unsure. Respondents gave a number of 
reasons why they would or would not continue to use this process. Reasons for 
continuing include: the process is very helpful (29%), focuses attention (4%), produces 
results (4%), and validates existing processes (4%). Reasons for being unsure about 
continuing include: continuation depends upon administration leadership (8%), 
needing community resources (4%), and a lack of staff development time (4%). 

Participants were also positive about their ability to use this benchmarking 
process with other changes they are making in -heir school. A majority (83%) stated 
that they would be able use this process for other changes. No respondents said they 
would not use this process, and 10"/o were unsure. The predominant reason they 
would use this process in other area;, was that it was a sensible, practical, and useful 
process (42%). Some felt they would be nb.o to use it because it was adaptable (11%). 
Others expressed reservations, stating that they would be able to use this process 
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only if more administrators and teachers were involved and followed-through on the 
results (26%). 

These survey results agree with facilitators' observations and responses from 
the formative surveys. In general, participants found this process to be quite helpful 
in improving their understanding of Tech Prep within their school. Quite often it 
was obvious that there was not a common understanding of what Tech Prep was 
amongst team members, and this lack of a common philosophy often hindered their 
attempts to communicate with parents and students and recruit teachers. In 
particular, they found the Critical Success Factors and implementation checklist 
activities to be invaluable in structuring their dialog and helping them develop a 
common understanding. 

For a number of schools, the benchmarking process became a staff 
development activity. Team members in these schools used the checklist and other 
activities to assist their fellow teachers and administrators in their understanding of 
Tech Prep as it was implemented in their school. In many cases, a number of the 
stakeholders within a team were unaware at the start of the process of what Tech 
Prep was in their school and the extent to which it had or had not been implemented. 

Whereas the benchmarking process pointed out that a number of these schools 
had no common philosophy of Tech Prep, it also indicated a lack of specific 
definitions for many of the elements within Tech Prep. For example, team members 
in many schools were unsure as to what a career plan was, or what specifically was 
involved in an integrated and applied course. Even in those schools which had 
definitions for these elements, there was often as much discrepancy in definitions 
amongst team members as there was amongst schools. 

Data Collection 

Although the self-assessment process was successful the data collection and 
implementation profile aspects of the process met with considerable difficulty and 
resistance. Many of the participants viewed data collection suspiciously, and were 
concerned about the negative effects that the numbers could have. Others felt that it 
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was too much trouble to collect reliable and valid data. In many cases, institutional 
structures supported this latter belief, making it very difficult, and in some cases 
impossible, to access data. A widespread concern was a lack of definitions for the 
practices which they were collecting data on. Many practitioners were unsure of 
what to count under such Tech Prep practices as integrated and /or applied 
curriculum, career plan, and career cluster. Some of the specific concerns about data 
collection that were raised include: bias and lack of definition of the data being 
collected (48%), comparison and ranking of schools (24%), use of data for staff or 
budget reductions (15%), and concerns over accuracy (14%). 

The perception of the facilitators at the beginning of this process was that there 
was little understanding of the uses of data in program improvement. By the end of 
the process, however, it seemed that more participants had come to believe that data 
could be useful. When asked to rate the usefulness of the type of data collected in 
the Tech Prep Implementation Profile on a seven-point scale, 67% rated it 5 or above. 
The mean rating was 5.2. When asked what the uses of this type of information 
might be, 39% said program improvement, 35% said justification for funding and 
resources, and 26% said dissemination, marketing, and public relations. 

Participants were asked how likely they were to collect data when developing 
or changing their educational practices. Again using a seven-point scale, 70% rated 
themselves 5 or higher. The mean response was 4.9. Twenty five percent said that 
such data could be used to inform and validate change. Twenty five percent said 
that they have already done so, and 19% said that it needed to be done. Thirteen 
percent said that they did not know how, and 13% said that they did not have the 
time or resources to do so. 

Participants were also asked about the barriers to gathering and analyzing data 
which they perceive in their schools. These barriers include: lack of time (39%), lack 
of money and resources (17%), no definitions or guidelines on what to measure 
(15%), lack of awareness or interest in collecting data (10%), lack of knowledge and 
tools for collecting data (7%), and problems with the accuracy of data (7%). 
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Summary Discussion 

The benchmarking model laid out in this paper seeks to accommodate local 
practitioners' needs for information for use in program improvement, while providing 
policy makers with a way to collect the information they need to ensure program 
funds are spent appropriately. This is accomplished through self-assessment and 
data collection systems that are aligned with state-level performance measures and 
give local practitioners ownership of the data collected in their school. Ideally, local 
practitioners collect data on all of the various benchmarks specified above for their 
own program improvement purposes, and pass the information relevant to state-level 
performance measures on to relevant state agencies. 

The pilot test of this model validates the componential structure of the self- 
assessment and data collection tools. Local practitioners and state-level policy 
makers both agreed that this practice-focused approach met their requirements for a 
way to look at diverse programs without identifying a specific Tech Prep or STW 
track. Pilot site participants also found both the self-assessment and da^a collection 
tools useful for understanding the extent and nature of their Tech Prep 
implementation. 

Self- Assessment Tools 

Participants in this pilot lest found the self-assessment tools, particularly the 
Tech Prep Implementation Checklist, beneficial in the way they guided and enhanced 
their team's reflection on the. nature and level of implementation of Tech Prep 
practices in their school. They felt that this process helped develop a greater 
understanding of the full range of practices included under the Tech Prep umbrella 
and provided an excellent staff development opportunity which promoted staff 
understanding and collaboration. 

Use of these self-assessment tools also pointed out the need for common 
definitions both within and among schools implementing Tech Prep. Much of each 
team's discussion centered around differences in interpretation of various labels, such 
as "Career Major" and "Integrated and Applied Course." Quite often there was little 



agreement on what a given label referred to. This divergence was even greater 
across schools. Many of these labels meant very different practices in different 
schools. Furthermore, Tech Prep meant very different things in different schools In 
many cases, Tech Prep and STW were considered to be the same, referring to wide- 
scale secondary or K-12 reform efforts. In others, there was more of a focus on 
meeting the needs of non-college bound students. One school opted not to call its 
efforts Tech Prep or STW, instead focusing on career development practices through 
a Personal Development Plan." 

Data Collection Tools 

Participants were less entliusiastic about the data collection tools. While many 
saw the usefulness of data collection for program improvement, most were 
suspicious about the possibility that data could "fall into the wrong hands " From 
their point of view, data which highlighted areas in need of improvement could be 
used against them, either by district- or state-level administrators or in the media 
Positive results could also be used against them as a justification for budget or staff 
cuts. Many felt that there was a strong possibility of data being misinterpreted or 
misused, and that it should not be collected without strenuous safeguards on the 
distribution and interpretation of the results. 

It is this altitude toward data which is likely responsible for the difficulHes all 
SIX pilot sites had collating data for the benchmarks. In many respects, these schools 
were not institutionally and attitudinally equipped to collect and use this data in a 
reliable way. For the most part, it was difficult for practitioners to access data on 
their own programs and students. In some cases this data was not collected by the 
school. When it was collected, it was either in an unusable form or locked away in a 
file or database which they could not access. To work around this, some 
practitioners had to rely on last year s reports to the state Department of Public 
Instruction for their data. Furthermore, practitioners were unsure of what to include 
as Tech Prep practtces. This results fron, a lack of generally agreed upon definitions 
tor most of these practices. Together, these limitations place severe restrictions on the 
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reliability and ultimate usefulness of the data collected. 
Next Steps 

As a federally funded STWOA Implementation Grant state, Wisconsin is 
developing a longitudinal evaluation system for use state-wide. Similarities between 
many components of Tech Prep and STW allow for the use of this benchmarking 
system and its component structure for both program and student outcome analyses. 
As a taxonomy for local system analysis, looking at implementation, participation, 
and outcomes for each of the seven component areas provides the flexibility needed 
to adapt to diverse local systems and the expected growth within each system. 
Changes in the nature and level of implementation of these diverse STW practices 
can be accommodated easily under the qualitative and quantitative approach taken 
here. Similarly, student outcomes from many diverse schools can be tmderstood 
through a focus on the practices which lead to those outcomes. In this way, students 
who have received vastly different forms of STW education are not compared with 
each other. Rather, it is the practices which are compared, yielding information on 
which are most effective. 

In addition, the STWOA evaluation will provide an opportunity to expand the 
testing and refinement of the benchmarking model as a program improvement tool. 
Many of these schools have expressed interest in being involved in this process, and 
the model will certainly be improved as a result of this use. In this next round, 
significant attention will be directed toward the improvement of data collection 
systems and toward addressing the institutional and attitudinal difficulties 
encountered here. The hope is that this will result in a cohort of practitioners who 
are able to collect, understand, and use both qualitative and quantitative data for 
program improvement. 
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Conclusion 

The benchmarking model introduced in this paper is not meant to replace 
traditional forms of formative evaluation. Rather, the goal of benchmarking is to 
further local practitioners' program improvement efforts. In doing so, benchmarking 
supplements formative evaluation by creating a rationale for practitioners to collect 
reliable and valid data. It also allows practitioners to develop a better understanding 
of what it is they are doing and why, as well as understanding the accomplishments 
of others. By combining self-assessment and data collection with a focus on Tech 
Prep and STW practices, this benchmarking model will give these practitioners the 
information they need to improve the ways in which they educate students. In 
addition, the data collected through this process provides a link to summative 
evaluation. Because this data is collected on a regular basis, it provides both a 
baseline and a longitudinal source of meaningful information about practices and 
outcomes. 
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SCHOOL-TO-WORK TRANSITION: 

Issues and Strategies for Evaluation and Program Improvement 
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